Dynamic aperture positioning for stereo endoscopic cameras

ABSTRACT

A stereoscopic endoscope that includes at least one image sensor, one or more processing devices, and a display device. The image sensors sense a pair of stereo images, based on capturing light passing through apertures electronically defined at a first aperture location and a second aperture location, respectively, on a liquid crystal layer within the endoscope. The size of each aperture, spacing between the apertures, and polarization state associated with each aperture are controlled using corresponding control signals provided to the liquid crystal layer. The image data captured through the apertures is used to generate control signals representing one or more views of a surgical scene. The views are then presented on a display device.

PRIORITY CLAIM

This application is a continuation of U.S. application Ser. No.15/956,655, filed on Apr. 18, 2018, now U.S. Pat. No. 10,365,554, whichclaims priority to U.S. Provisional Application 62/652,826, filed onApr. 4, 2018. The entire contents of each of the foregoing applicationsare incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates to three-dimensional (3D) endoscopes that may beused in surgical systems.

BACKGROUND

Minimally invasive surgical systems are being developed to reduce thetrauma experienced by patients undergoing surgical interventions. Thesesystems require only small incisions and surgeons use stick like camerasand instruments to perform the procedure. In addition to reducingtrauma, teleoperated systems of this type increase a surgeon's dexterityas well as to allow a surgeon to operate on a patient from a remotelocation. Telesurgery is a general term for surgical systems where thesurgeon uses some form of remote control, e.g., a servomechanism, or thelike, to manipulate surgical instrument movements rather than directlyholding and moving the instruments by hand. In such a telesurgerysystem, the surgeon is provided with an image of the surgical sitethrough a display device. Based on visual feedback received through thedisplay device, the surgeon performs the surgical procedures on thepatient by manipulating master control input devices, which in turncontrol the motion of tele-robotic instruments.

SUMMARY

The technology described herein describes stereoscopic cameras withcontrollable pupil locations. The controllable pupil locations allow astereoscopic view that adapts to the axial rotation of an endoscope withan angled view. Methods are described as to how to control the locationof the pupils to create a natural stereo view during scope rotation orduring head motion by a viewer. The device is electronic and may becontrolled through software.

In one aspect, this document features a method for generating a view ofa scene, the method including determining, by one or more processingdevices, an angle of orientation defined by a line connecting a firstaperture location and a second aperture location of a stereoscopicendoscope with respect to a reference orientation. The method alsoincludes adjusting at least one of the first and second aperturelocations, while maintaining a spacing between the first and secondaperture locations, to maintain the angle of orientation across multipleendoscope orientations. The method also includes creating an aperture ateach of the first and second aperture locations, and generating arepresentation of the view for presentation on a display deviceassociated with the stereoscopic endoscope using signals (representingimage data) based on light captured through the apertures created at thefirst and second aperture locations.

In another aspect, this document describes a system that includes astereoscopic endoscope, a display device, and one or more processingdevices. The stereoscopic endoscope includes at least one image sensorfor sensing a first image and a second image of a pair of stereo images.The first and second images are sensed based on light passing throughapertures electronically defined at a first aperture location and asecond aperture location, respectively, on a liquid crystal layer withinthe endoscope. The one or more processing devices are configured todetermine an angle of orientation defined by a line connecting the firstaperture location and the second aperture location of the stereoscopicendoscope with respect to a reference orientation, adjust at least oneof the first and second aperture locations, while maintaining a spacingbetween the first and second aperture locations, to maintain the angleof orientation across multiple endoscope orientations, and create anaperture at each of the first and second aperture locations. The one ormore processing devices are also configured to generate a representationof views of a surgical scene using the pair of stereo images capturedthrough the apertures created at the first and second aperturelocations. The display device is configured to presenting therepresentation of the views.

In another aspect, this document describes one or more machine-readablestorage devices having encoded thereon computer readable instructionsfor causing one or more processing devices to perform variousoperations. The operations include determining an angle of orientationdefined by a line connecting a first aperture location and a secondaperture location of a stereoscopic endoscope with respect to areference orientation, and adjusting at least one of the first andsecond aperture locations, while maintaining a spacing between the firstand second aperture locations, to maintain the angle of orientationacross multiple endoscope orientations. The operations also includecreating an aperture at each of the first and second aperture locations,and generating a representation of a view for presentation on a displaydevice associated with the stereoscopic endoscope using signals based onlight captured through the apertures created at the first and secondaperture locations.

Implementations of the above aspects can include one or more of thefollowing features. The reference orientation can be perpendicular tothe direction of earth's gravity. Information indicative of anorientation of the head of a user operating the stereoscopic endoscopecan be received, and the reference orientation can be determined inaccordance with the information indicative of the orientation of thehead of the user. Adjusting at least one of the first and secondaperture locations can include selecting locations of a pair of liquidcrystal display (LCD) segments from a set of LCD segments disposed in asubstantially annular configuration in an optical path of thestereoscopic endoscope. Creating the apertures at each of the first andsecond aperture locations can include controlling a first LCD segment inthe pair of LCD segments such that the first LCD segment changes to astate in which the first LCD segment allows more light to pass throughas compared to a different, relatively dark state, and controlling asecond LCD segment in the pair of LCD segments such that the second LCDsegment changes to a state in which the second LCD segment allows morelight to pass through as compared to a different, relatively dark state.The first and second LCD segments can be controlled to acquire a firstimage and a second image, respectively, substantially concurrently.Light passing through the first LCD segment can pass through a firstpolarizer, and light passing through the second LCD segment can passthrough a second polarizer that polarizes light differently from thefirst polarizer. The first polarizer can be orthogonal with respect tothe second polarizer. The light passing through the apertures created atthe first and second aperture locations can be sensed using a firstsensor and a second sensor, respectively, the first and second sensorsbeing disposed on two opposing sides of a polarizing beam splitter. Theapertures can be created at the first and second aperture locations in asequential pattern. The light passing through the apertures created atthe first and second aperture locations can be sensed using a singlesensor. The representation of the view may be presented on the displaydevice. Responsive to presenting the representation of the view on thedisplay device, user input pertaining to operating a surgical device ata surgical scene may be received.

In another aspect, this document features a stereoscopic endoscope thatincludes at least one image sensor for sensing a first image and asecond image of a pair of stereo images. The first image is sensed basedon light passing through a first aperture within the endoscope, and thesecond image is sensed based on light passing through a second aperturewithin the endoscope. A liquid crystal layer disposed between two layersof glass includes a first arrangement of electrodes, such that each ofthe first aperture and the second aperture is created in the liquidcrystal layer using a portion of the first arrangement of electrodes.

Implementations can include one or more of the following features. Thelight passing through the first aperture can be polarized differently ascompared to the light passing through the second aperture. Thestereoscopic endoscope can include a first image sensor, a second imagesensor, and an optical element that directs incident light to the firstimage sensor or the second image sensor based on polarization state ofthe incident light. Each of the first aperture and the second aperturecan be controlled to be located at various locations on the liquidcrystal layer in response to corresponding control signals providedthrough the first arrangement of electrodes. Locations of the first andsecond apertures can be controllable in accordance with an orientationof the endoscope with respect to a reference orientation. Thestereoscopic endoscope can include a first image sensor and a secondimage sensor, wherein the first image and the second image are sensed bythe first image sensor and the second image sensor, respectively,substantially concurrently. The first image and the second image can besensed by a single image sensor sequentially. The stereoscopic endoscopecan include a first portion housing a front end lens assembly, and asecond portion including an elongated shaft that houses the liquidcrystal layer and the at least one image sensor. The second portion canbe disposed at an angle with respect to the first portion. The angle canbe one of: 0°, 30°, and 45°. A location of at least one of the first andsecond apertures can be electronically adjusted using the firstarrangement of electrodes to maintain an angle between (i) a lineconnecting the first and second apertures, and (ii) a referenceorientation. The angle between (i) the line connecting the first andsecond apertures, and (ii) the reference orientation can be maintainedwhile also maintaining a spacing between the first and second apertures.The angle between (i) a line connecting the first and second apertures,and (ii) the reference orientation can be maintained using one or morecontrol signal calculated based on one or more previously capturedimages.

In another aspect, this document features a system that includes astereoscopic endoscope, a display device, and one or more processingdevices. The stereoscopic endoscope includes at least one image sensorfor sensing a first image and a second image of a pair of stereo images.The first and second images are sensed based on light passing throughapertures electronically defined at a first aperture location and asecond aperture location, respectively, on a liquid crystal layer withinthe endoscope. The one or more processing devices are configured todetermine an angle of orientation defined by a line connecting the firstaperture location and the second aperture location of the stereoscopicendoscope with respect to a reference orientation, and adjust at leastone of the first and second aperture locations, while maintaining aspacing between the first and second aperture locations, to maintain theangle of orientation across multiple endoscope orientations. The one ormore processing devices are also configured to create an aperture ateach of the first and second aperture locations, and generaterepresentation of views using the pair of stereo images captured throughthe apertures created at the first and second aperture locations. Thedisplay device is configured for presenting the representation of theviews.

In another aspect, this document features one or more machine-readablestorage devices having encoded thereon computer readable instructionsfor causing one or more processing devices to perform variousoperations. The operations include determining an angle of orientationdefined by an orientation of a line connecting a first aperture locationand a second aperture location of a stereoscopic endoscope relative to areference orientation, and adjusting at least one of the first andsecond aperture locations, while maintaining a spacing between the firstand second aperture locations, to maintain the angle of orientation, inresponse to movement of the stereoscopic endoscope. The operationsfurther include capturing image data through apertures created at thefirst and second aperture locations.

In another aspect, this document features a method for generating a twoviews of a scene, the method comprising, generating two or more regionsof a first liquid crystal layer for allowing polarized light to passthrough such regions, and allowing said light to pass through apolarizer. The method also includes allowing said light to pass througha second liquid crystal layer, and separating said light, on the basisof polarization, into substantially two portions, each portion includinglight of predominantly one polarization state.

In another aspect, this document describes a stereo endoscope thatincludes a two pupil optical system wherein each pupil is orthogonallypolarized, and wherein light from the polarized pupils are split ontotwo image sensors based on polarization.

In another aspect, this document features a stereo endoscope thatincludes a two pupil optical system wherein each pupil is orthogonallypolarized, and wherein light from the polarized pupils casting onto asingle image sensor with a checkerboard of pixels sensitive toorthogonal polarization states.

Some or all of the embodiments described herein may provide one or moreof the following advantages. By providing a split-pupil endoscope camerain which the pupil locations are controlled using liquid crystal displayelements, an electronically controllable imaging apparatus with few, ifany, mechanically moving parts may be implemented within thespace-constrained and/or resource constrained environment of anendoscope. The pupil locations may be controlled based on informationassociated with the orientation of a surgeon's head. Other controlinputs may be used as well and may facilitate certain image processingrequirements. This in turn can keep the two pupils separated by apredetermined distance along a reference direction even when the camerais rotated during the operation of an endoscope. The direction ofseparation of the pupils relative to gravity for example may becontrolled at will. Maintaining such predetermined distance between thestereoscopic pupils may allow for displaying accurate 3D representationsfor different orientations of the endoscope. In some cases, suchrepresentations may be consistent with the natural way a surgeon wouldview the corresponding surgical scene, and therefore may contribute toimproving the user-experience for the surgeon. The separation distanceof the pupils (or inter pupillary distance) may also be controlled ifdesired.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of an example patient-side cart of acomputer-assisted tele-operated surgery system.

FIG. 2 is a front view of an example surgeon console of acomputer-assisted tele-operated surgery system.

FIG. 3 is a side view of an example robotic manipulator arm assembly ofa computer-assisted tele-operated surgery system.

FIG. 4 is a schematic diagram illustrating optical paths through anexample of a 30° stereoscopic endoscope.

FIGS. 5A and 5B are schematic representations of examples of polarizersusable with the technology described herein.

FIGS. 6A and 6B are examples of optical masks usable with the technologydescribed herein.

FIGS. 7A and 7B are example arrangements of electrodes usable in a firstportion of a multi-pupil device in accordance with one implementation ofthe technology described herein.

FIG. 8A is an example arrangement of electrodes usable in a secondportion of the multi-pupil device of FIG. 7A.

FIG. 8B is an example of rotator segments created with the combinationof electrode arrangements of FIGS. 7A, 7B, and 8A.

FIG. 8C is an example of the electrode used in conjunction with those in8A to create patterns of polarization control as illustrated in 8B.

FIG. 9A is another example arrangement of electrodes usable in a secondportion of the multi-pupil device of FIG. 7A.

FIG. 9B is an example of rotator segments created with the combinationof electrode arrangements of FIGS. 7A, 7B, and 9A

FIG. 9C is an example of the electrode used in conjunction with those in9A to create patterns of polarization control as illustrated in 9B

FIGS. 10A and 10B are examples of pupil positions for two differentorientations of a stereoscopic endoscope, respectively.

FIGS. 10C and 10D illustrate examples of different configurations andpositions of the pupils.

FIG. 11 is a schematic diagram illustrating optical paths through anexample of a 0° stereoscopic endoscope with two sensors.

FIG. 12A is a schematic diagram illustrating optical paths through anexample of a 0° stereoscopic endoscope that uses one sensor for sensingboth stereoscopic images substantially simultaneously.

FIG. 12B is a schematic diagram illustrating optical paths through anexample of a 30° stereoscopic endoscope that uses one sensor for sensingboth stereoscopic images.

FIG. 13 is a schematic diagram illustrating a process of sensing bothstereoscopic images using a single sensor with a rolling shutter.

FIG. 14 is a flowchart of an example process for generating a 3Drepresentation using technology described herein.

DETAILED DESCRIPTION

This document describes technology that facilitates automatic correctionof camera aperture positions in stereoscopic endoscopes such that theaperture positions remain substantially fixed with respect to areference frame even when the endoscope is oriented at different anglesduring a surgery. In some implementations, this can allow a more naturalperception of the 3D representation of the surgical scene as presentedvia stereo images on a surgeon's console. For example, the surgeon mayvisualize the surgical scene more naturally (e.g., without having totilt or reorient her head) even when the endoscope is oriented atarbitrary angles. In some cases, this can improve the overalluser-experience for the surgeon during surgical procedures. By allowingfor aperture positions to be controlled electronically, potentially withfew or no moving mechanical parts, the technology described hereinfacilitates implementations suited to space constrained and/or resourceconstrained environments of stereoscopic endoscopes. Endoscope in thiscontext may be a rigid device that incorporates optics and one or moreimage sensors to create a camera system; a flexible device that has awristed section; a flexible device with a camera at the distal end; anoptical endoscope (sometimes referred to as a Hopkins endoscope) with acamera, or a similar device.

Aspects of the technology are described primarily in terms of animplementation using da Vinci® surgical systems developed by IntuitiveSurgical, Inc. of Sunnyvale, Calif. Examples of such surgical systemsare the da Vinci® Xi™ Surgical System (Model IS4000). It should beunderstood that aspects disclosed herein may be embodied and implementedin various ways, including computer-assisted, non-computer-assisted, andhybrid combinations of manual and computer-assisted embodiments andimplementations. Implementations on da Vinci® Surgical Systems, e.g. theModel IS4000 are described for illustrative purposes, and are not to beconsidered as limiting the scope of the inventive aspects disclosedherein. As applicable, inventive aspects may be embodied and implementedin both relatively smaller, hand-held, hand-operated devices andrelatively larger systems that have additional mechanical support, aswell as in other embodiments of computer-assisted tele-operated medicaldevices. While the technology is described primarily with reference toan example of a peer-in display, the technology may also be used inother types of wearable or non-wearable display devices such as ahead-mounted display device used, for example, in virtual or augmentedreality (VR/AR) systems. The images captured may also be displayed on alarge format display such as a 3D TV device, projected onto a screen, orviewed by a user wearing stereo glasses. Alternatively, an auto-stereotype display may be used. Examples of an auto-stereo device include alenticular liquid crystal display (LCD) that may also incorporate headand or eye tracking of the viewer (user).

Referring to FIGS. 1 and 2, systems for minimally invasivecomputer-assisted telesurgery (also referred to as MIS) can include apatient-side cart 100 and a surgeon console 50. Telesurgery is a generalterm for surgical systems where the surgeon uses some form of remotecontrol, e.g., a servomechanism, or the like, to manipulate surgicalinstrument movements rather than directly holding and moving theinstruments by hand. The robotically manipulatable surgical instrumentscan be inserted through small, minimally invasive surgical apertures totreat tissues at surgical sites within the patient body, avoiding thetrauma associated with rather large incisions required for open surgery.These robotic systems can move the working ends of the surgicalinstruments with sufficient dexterity to perform quite intricatesurgical tasks, often by pivoting shafts of the instruments at theminimally invasive aperture, sliding of the shaft axially through theaperture, rotating of the shaft within the aperture, and/or the like.

In the depicted embodiment, the patient-side cart 100 includes a base110, a first robotic manipulator arm assembly 120, a second roboticmanipulator arm assembly 130, a third robotic manipulator arm assembly140, and a fourth robotic manipulator arm assembly 150. Each roboticmanipulator arm assembly 120, 130, 140, and 150 is pivotably coupled tothe base 110. In some embodiments, fewer than four or more than fourrobotic manipulator arm assemblies may be included as part of thepatient-side cart 100. While in the depicted embodiment, the base 110includes casters to allow ease of mobility, in some embodiments thepatient-side cart 100 is fixedly mounted to a floor, ceiling, operatingtable, structural framework, or the like.

In a typical application, two of the robotic manipulator arm assemblies120, 130, 140, or 150 hold surgical instruments and a third holds astereo endoscope. The remaining robotic manipulator arm assembly isavailable so that a third instrument may be introduced at the work site.Alternatively, the remaining robotic manipulator arm assembly may beused for introducing a second endoscope or another image-capturingdevice, such as an ultrasound transducer, to the work site.

Each of the robotic manipulator arm assemblies 120, 130, 140, and 150 isconventionally formed of links that are coupled together and manipulatedthrough actuatable joints. Each of the robotic manipulator armassemblies 120, 130, 140, and 150 includes a setup arm and a devicemanipulator. The setup arm positions its held device so that a pivotpoint occurs at its entry aperture into the patient. The devicemanipulator may then manipulate its held device so that it may bepivoted about the pivot point, inserted into and retracted out of theentry aperture, and rotated about its shaft axis.

In the depicted embodiment, the surgeon console 50 includes astereoscopic peer-in display 45 so that the user may view the surgicalwork site in stereo vision from images captured by the stereoscopiccamera used in conjunction with the patient-side cart 100. Left andright eyepieces, 46 and 47, are provided in the stereoscopic peer-indisplay 45 so that the user may view left and right display screensinside the display 45 respectively with the user's left and right eyes.While viewing typically an image of the surgical site on a suitableviewer or display, the surgeon performs the surgical procedures on thepatient by manipulating master control input devices, which in turncontrol the motion of robotic instruments.

The surgeon console 50 also includes left and right input devices 41, 42that the user may grasp respectively with his/her left and right handsto manipulate devices (e.g., surgical instruments) being held by therobotic manipulator arm assemblies 120, 130, 140, and 150 of thepatient-side cart 100 in preferably six or more degrees-of-freedom(“DOF”). Foot pedals 44 with toe and heel controls are provided on thesurgeon console 50 so the user may control movement and/or actuation ofdevices associated with the foot pedals.

A processing device 43 is provided in the surgeon console 50 for controland other purposes. The processing device 43 performs various functionsin the medical robotic system. One function performed by processingdevice 43 is to translate and transfer the mechanical motion of inputdevices 41, 42 to actuate their corresponding joints in their associatedrobotic manipulator arm assemblies 120, 130, 140, and 150 so that thesurgeon can effectively manipulate devices, such as the surgicalinstruments. Another function of the processing device 43 is toimplement the methods, cross-coupling control logic, and controllersdescribed herein.

The processing device 43 can include one or more processors, digitalsignal processors (DSPs), field-programmable gate arrays (FPGAs), and/ormicrocontrollers, and may be implemented as a combination of hardware,software and/or firmware. Also, its functions as described herein may beperformed by one unit or divided up among a number of subunits, each ofwhich may be implemented in turn by any combination of hardware,software and firmware. Further, although being shown as part of or beingphysically adjacent to the surgeon console 50, the processing device 43may also be distributed as subunits throughout the telesurgery system.One or more of the subunits may be physically remote (e.g., located on aremote server) to the telesurgery system.

Referring also to FIG. 3, the robotic manipulator arm assemblies 120,130, 140, and 150 can manipulate devices such as an endoscopic stereocamera and surgical instruments to perform minimally invasive surgery.For example, in the depicted arrangement the robotic manipulator armassembly 120 is pivotably coupled to an instrument holder 122. A cannula180 and a surgical instrument 200 are, in turn, releasably coupled tothe instrument holder 122. The cannula 180 is a hollow tubular memberthat is located at the patient interface site during a surgery. Thecannula 180 defines a lumen in which an elongated shaft 220 of theendoscopic camera (or endoscope) or surgical instrument 200 is slidablydisposed. As described further below, in some embodiments the cannula180 includes a distal end portion with a body wall retractor member. Theinstrument holder 122 is pivotably coupled to a distal end of therobotic manipulator arm assembly 120. In some embodiments, the pivotablecoupling between the instrument holder 122 and the distal end of roboticmanipulator arm assembly 120 is a motorized joint that is actuatablefrom the surgeon console 50 using the processing device 43.

The instrument holder 122 includes an instrument holder frame 124, acannula clamp 126, and an instrument holder carriage 128. In thedepicted embodiment, the cannula clamp 126 is fixed to a distal end ofthe instrument holder frame 124. The cannula clamp 126 can be actuatedto couple with, or to uncouple from, the cannula 180. The instrumentholder carriage 128 is movably coupled to the instrument holder frame124. More particularly, the instrument holder carriage 128 is linearlytranslatable along the instrument holder frame 124. In some embodiments,the movement of the instrument holder carriage 128 along the instrumentholder frame 124 is a motorized, translational movement that isactuatable/controllable by the processing device 43. The surgicalinstrument 200 includes a transmission assembly 210, the elongated shaft220, and an end effector 230. The transmission assembly 210 may bereleasably coupled with the instrument holder carriage 128. The shaft220 extends distally from the transmission assembly 210. The endeffector 230 is disposed at a distal end of the shaft 220.

The shaft 220 defines a longitudinal axis 222 that is coincident with alongitudinal axis of the cannula 180. As the instrument holder carriage128 translates along the instrument holder frame 124, the elongatedshaft 220 of the surgical instrument 200 is moved along the longitudinalaxis 222. In such a manner, the end effector 230 can be inserted and/orretracted from a surgical workspace within the body of a patient.

Laparoscopic surgery can entail the surgeon viewing the surgical sitewith the endoscope and performing fine motor manipulations withlaparoscopic instruments for exploration, dissection, suturing, andother surgical tasks. These tasks often require fine bi-manualinteractions with tissue. In some cases, such bi-manual motor tasks maygenerally be more easily performed when the surgeon is presented with a3D view of the surgical scene. The surgical workspace within the body ofa patient (the surgical scene) can be presented as a 3D visualization tothe surgeon via the stereoscopic display 45. While the technologydescribed herein primarily uses examples of a peer-in stereoscopicdisplay, other types of stereoscopic and non-stereoscopic displays arealso within the scope of the technology. A peer-in stereoscopic displayrefers to a display that allows a user to look into the display withouthaving to wear it or simultaneously share it with another user. A stereomicroscope can be an example of a peer-in stereoscopic device. Thestereoscopic display 45, as illustrated in FIG. 2 is another example ofa peer-in stereoscopic display.

In some implementations, a peer-in stereoscopic display 45 can includetwo display screens (one for each eye) each of which displays one of twoimages corresponding to a stereo pair images captured using astereoscopic endoscope. Such stereo pair images, when displayed, causesa user to perceive depth in the displayed images. Stereo pair images canbe captured in various ways including, for example, using two spatiallyseparated cameras in a “dual camera” approach, collecting a light fieldin a plenoptic camera approach, or as described here in, using a dualpupil type device where two spatially separated apertures (also referredto as pupils) cause the capture of two slightly different images fromtwo slightly different perspectives.

In the dual camera approach, two separate cameras and associated opticscan be oriented such that they capture substantially the same scene fromtwo locations separated by an ‘inter pupillary distance’. This approachmimics human vision where two spatially separated eyes are used tocapture a 3D view of the world. The orientation of the camera pair, atthe time of acquisition, determines the orientation at which the imagesmay be reproduced. For example, the cameras may be separated along ahorizontal axis such that a view corresponding to a straight (or‘normal’) head or an inverted head may be captured. However, forarbitrary angles of orientation of the two cameras, the horizontaldistance between the cameras varies, thereby making the 3D presentationchallenging. A camera pair with cameras facing the horizon separated ina plane perpendicular to gravity has fixed pupils and when rotated 90degrees captures one image from above the other; this is not always thedesired behavior.

In another approach, 3D scenes may be captured by sampling light fields,for example, using a plenoptic camera that captures both intensity anddirection of some fraction of the many rays that pass through an opticalsystem. Plenoptic cameras can include, for example, camera arrays,movable cameras, or cameras that contain lens arrays to sample the lightfield at some spatial and angular resolution. In this approach, it ispossible to reconstruct, post-acquisition, synthetic camera views inwhich the depth of field may be manipulated to control the focus atdifferent portions of the scene. Multi-view images may also be createdin which the camera appears to move a small amount relative to the scenethus showing the depth in the acquired scene. These plenoptic systemsmay simulate the two views of a stereo pair by selecting the appropriatepencils of rays. Typically however, there is a substantial reduction inresolution in such systems that is undesired, as the information frommost pixels is not used so those pixels are wasted.

In some implementations of a dual pupil type device, a single opticalsystem can be used for both eyes. In some cases, a mechanical aperturecan be moved from one location to another (e.g., from right to left) toallow light pass through two locations sequentially. If this is done ata sufficiently high rate, say 120 Hz, then a smooth sequence of stereoimages may be acquired with a suitable high-speed camera. Instereoscopic endoscopes though, where the constraints on space and powerresources can limit mechanical movements, implementing such a mechanicalaperture may be challenging. The technology described herein providesfor electronically controllable apertures that substantially avoids thedrawbacks of mechanical apertures to realize a camera system suitablefor incorporation in an endoscope. Further, by adjusting the location ofthe apertures based on an orientation of the endoscope, the technologyfacilitates 3D visualizations that may be perceived as natural, andthereby substantially improves the user experience of the surgeon duringa surgical process. Using the technology described herein, imagesequences are acquired using electronic shutters (e.g., based onactivating/deactivating liquid crystal display (LCD) segments topass/block light in an optical path) that may be synchronized to thedisplay. This can result in the left eye observing the view through onepupil and the right eye observing the view through the other. Such splitpupil cameras may therefore represent a simple light field camera inwhich only two views are sampled, and each at full resolution with nowasted pixels.

In some implementations, a stereo endoscope can include optics that areconfigured to look straight out the end of a tube or shaft housing aportion of the endoscope. Such endoscopes may be referred to as 0°endoscopes. However, in some cases, surgeons may also need endoscopesthat look ‘down’ on their work area. For example, in laparoscopicsurgery, this view is typically oriented at 30° with respect to theshaft of the endoscope. Endoscopes with other orientations (e.g., suchas ones used in arthroscopy) are also possible; angles of view such as45 and 70 degrees are also common.

FIGS. 4 and 11 illustrate examples of a 30° endoscope and a 0°endoscope, respectively. Specifically, FIG. 4 is a schematic diagramillustrating optical paths of the invention through an illustrativesimplified 30° stereoscopic endoscope 400, and FIG. 11 is a schematicdiagram illustrating optical paths through an example of a 0°stereoscopic endoscope 1100. Referring to FIG. 4, the endoscope 400includes an optical arrangement 401 of a plurality of lenses and otheroptical elements that direct light from a target imaging region 403 ontotwo image sensors 402 and 404, configured to sense the right view andthe left view, respectively. In some implementations, the opticalarrangement 401 includes a beam splitter 406 that allows a set of rays408 to pass through the surface 410 while reflecting the set of rays 412towards the image sensor 404. The surface 410 can be configured toreflect rays of one polarization state while allowing rays of adifferent polarization state to pass through. For example, the surface410 can include a wire grid layer (such as developed by Moxtek Inc. ofUT, USA, or Meadowlark Optics of CO, USA), which separates differentlinear polarization states through reflection and transmission. In someimplementations, this facilitates acquiring the left and right images ofthe stereo endoscope substantially simultaneously in a compact formfactor that avoids using larger diameter optics as may be needed forseparate physical optical channels for the left and right images. One ofthe advantageous aspects of the technology described herein is that theleft and right eyes match in performance as they use the same opticalpath. For example, if the optics 450 are moved longitudinally to focusthe images on sensors 402 and 404, the left and right eyes willinherently match in terms of their respective focus curves.Additionally, aberrations due to tolerances in the optical system willalso match between the images on the two sensors. This is beneficial formaintaining stereoscopic image quality.

In some implementations, the front end of the optical arrangement 401can include a sapphire window 414, a front negative lens 416, a firstportion 418 of a 30° prism, and a second portion 420 of the 30° prism.The 30° prism can include an air gap at the interface between the firstportion 418 and second portion 420 such that light reflected from thesurface of the second portion 420 is total internally reflected at theinterface between the first portion 418 and second portion 420, as shownusing the ray paths in FIG. 4. Some implementations may exclude thesapphire window 414, which is disposed to protect other portions of theoptical arrangement 401.

In some implementations, the optical arrangement 401 includes twoseparate pupils or apertures defined in order to pass lightcorresponding to a right view and a left view, respectively. The twoapertures may be defined by a combination of multiple optical elementssuch as one or more image transmission layers (e.g., liquid crystallayers), electrodes, and polarizers such that the apertures may begenerated and blocked electronically, allowing for dynamic control overthe location of the apertures. In some implementations, the aperturescan be defined using a first polarizer 422, a first liquid crystal layer424, a second polarizer 426 that is orthogonal to the first polarizer422, and second liquid crystal layer 428. The first liquid crystal layer424 can be energized using transparent electrodes disposed in a glasslayer 430 adjacent to the liquid crystal layer 424, and complementaryelectrodes disposed in a glass layer 431 located on the opposite side ofthe liquid crystal layer 424 as compared to the glass layer 430.Similarly, the second liquid crystal layer 428 can be energized usingtransparent electrodes disposed in a glass layer 432 adjacent to theliquid crystal layer 428 and complementary electrodes disposed in aglass layer 433 located on the opposite side of the liquid crystal layer428 as compared to the glass layer 432. The segments formed in liquidcrystal layer 424 can be energized (as described below in more details)to form a right pupil or aperture 434 that allows light corresponding tothe right view to pass through. The segments formed in liquid crystallayer 428 can be energized to define another region 436 that polarizeslight from the pupil 434 in a way that it is transmitted through, aslight 408, to the surface 410 of the polarizing beam splitter 406. Thesegments formed in liquid crystal layer 424 can also be energized toform the left pupil or aperture 438 that allows light corresponding tothe left view to pass through. Correspondingly, the segments formed inliquid crystal layer 428 can be energized to define another region 440that polarizes light from the pupil 438, light 412, in a way that it isreflected from the surface 410 of the polarizing beam splitter 406towards the image sensor 404. The unwanted light (e.g., light that doesnot pass through the desired apertures may be absorbed by masks or thesecond polarizer 426). This description primarily uses a liquid crystallayer as an example of an image transmission layer. However, otherelectronically controllable image transmission layers (e.g., one thatuses E-ink segments instead of liquid crystal segments) are within thescope of this disclosure.

In some implementations, the liquid crystal element 428 rotates thelight passing through the pupils 434 and 438 such that the light passingthrough the rotator pupil 436 is orthogonally polarized with respect tothe light passing through the rotator pupil 440. In someimplementations, the liquid crystal layer 424 can be segmented radiallyabout an annular region and one or more of the segments may be used todefine the pupil areas. The light emanating from both pupils defined onthe liquid crystal layer 424 has the same polarization and traversespolarizer 426. Light not in the pupil areas is not rotated and is thussubject to the extinction ratio of two crossed polarizers 422 and 426.Example schematic (or symbolic) representations of such polarizers areshown in FIGS. 5A and 5B, respectively. In some cases, one could reversethese (possibly at the expense of a change in contrast ratio) and havepolarizers 422 and 426 parallel to one another, and rotate thenon-desired light.

Referring to FIG. 5A, in some implementations, the orientation of thepolarizer 422 can be configured in accordance with the light sourceilluminating the surgical scene. For example, if the light sourceilluminating the surgical scene is polarized, the polarization directionand orientation of the polarizer 422 may be configured to be orthogonalto that of the light source, for example, to reduce the salience ofspecular reflections from the tissue surface. In some implementations,the polarizer 422 may incorporate a quarter wave plate together with acorresponding quarter wave plate on the illumination source to providean orthogonality in circular polarization between the illumination andthe imaging paths. In some cases, the polarizer 422 and or 426 mayinclude or be augmented with compensation films in a similar fashion tothose used on LCD displays to improve the angular performance and thusimproving the system's contrast ratio by improving the transmission andextinction ratios of the liquid crystal pupil forming assembly.

Referring to FIG. 5B, the polarizer 426 can be configured to transmitlight which is polarized appropriately and corresponds to the pupilareas defined using the liquid crystal layers. For example, thepolarizer 426 can be configured to transmit light passing through theareas appropriately energized using the electrodes on the glass layers430 and 432. Light passing through the non-energized areas may beabsorbed, for example, by the combination of the polarizers 422 and 426,and one or more masking layers disposed to absorb the light. In someimplementations, the polarization orientation of the polarizer 422 canbe orthogonal to that of the polarizer 426. However, other relativepolarization orientations of the two polarizers are also possible—forexample, the “transmission” and “blocking” regions can be reversedand/or the polarizers parallel instead of crossed but thoseconfigurations are generally less optimal. In some implementations, theorientations can be dependent on the degree of retardation the liquidcrystal layer 424 imparts.

In some implementations, and preferably, a black mask may be disposedwithin the optical arrangement 401 to restrict light from passingthrough areas where a pupil is not going to be defined by the use casesof the device. An example of such a mask 600 is illustrated in FIG. 6A.In some implementations, the mask 600 can be formed by metal deposition,thin metallic masks, formed by semiconductor like processes, padprinted, or otherwise formed either on a glass surface or on thepolarizer 422. In some implementations, the mask 600 may be implementedas a separate thin film (e.g. metal or Mylar) or glass plate. In theexample shown in FIG. 6A, the area 605 is the transparent area and thearea 603 is opaque or minimally transmissive. In this example, the area607 is also opaque or minimally transmissive in accordance with thenature of the pupils. In some implementations, the diameter of the area607 may be varied based on the pupil designs, and in some instances maybe zero. In some implementations, the area 603 may be implemented on onelayer (e.g., on the polarizer 422, or as an independent layer) and thearea 607 may be implemented on a subsequent layer (e.g., the glass layer430). These mask areas may be implemented in opaque thin foils or bydeposition of chrome or like materials or by pad printing an ink orother or substantially black material in a thin layer.

In some implementations, the mask may also be located adjacent to therotator pupils 436 and 440. In some implementations, a second maskadjacent to the rotator pupils can be used in conjunction with the mask600. An example of such a mask 620 is shown in FIG. 6B. Because theremay be a glass layer 432 between the pupils 434, 438 defining the liquidcrystal shutters and the subsequent rotator pupils 436, 440, the rotatorpupils 436, 440 can made be larger in area (as compared to the pupils434, 438) to account for the angle of the light passing through thepupils. Accordingly, the mask 620 (FIG. 6B) can have a transparentregion 625 that is larger than that of area 605 (FIG. 6A) to account forthe light spreading between the two liquid crystal layers 424 and 428.Transparent region 625 is defined by dark areas 623 and 627. In someimplementations, region 627 may not be present or be of vanishingdiameter. In some implementations, compensation films may be used toincrease the contrast ratio in resulting images.

In some implementations, instead of having two liquid crystal layers,the two apertures or pupils in a single layer may be polarizeddifferently. For example, a polarizing element may be disposed before,at, or after each aperture in the optical path such that the twopolarizing elements for the two pupils are orthogonal to each other. Insome implementations, the two polarizing elements can be linearpolarizers that are orthogonal to one another. In some implementations,the polarizing elements can include circular polarizers or colored notchfilters.

FIGS. 7A and 7B are example arrangements of transparent electrodesusable to control the pupil locations in the liquid crystal layer 424.Specifically, the FIG. 7A illustrates an example arrangement oftransparent conductive electrodes disposed on the glass layer 430. Theelectrodes may be disposed on the surface of glass layer 430 thatinterfaces with the liquid crystal layer 424, and can be configured tocontrol LCD segments forming the shutter pupils 434 and 438. The exampleof FIG. 7A shows a radial pattern of electrodes; with neighboringelectrodes 711 and 713 being two of a plurality of electrodescontrolling corresponding LCD segments defined in the liquid crystallayer 424. The neighboring electrodes are independent and areelectrically isolated from one another. The electrodes can be coupledwith electronic circuitry (not shown) using, for example, electricalcontacts disposed near the edge of the glass layer 430. The electroniccircuitry can be configured to drive the electrodes toactivate/deactivate or energize/deenergize corresponding LCD segments.

FIG. 7B, illustrates an example arrangement of transparent conductiveelectrode disposed in the glass layer 431. The electrode may be disposedon the surface of glass layer 431 that interfaces with the liquidcrystal layer 424, and in conjunction with the electrodes shown in FIG.7A, can be configured to control LCD segments forming the shutter pupils434 and 438. In the example shown in FIG. 7B, the electrode 717 isconnected to the central region 719 that interfaces with the LCDsegments in the liquid crystal layer 424. The electrode 717 can beconnected to the electronic circuitry configured to controlactivation/deactivation of the LCD segments. In some implementations, atleast a portion of the electronic circuitry may be disposed on the glasslayer 431. The order of the electrodes may be reversed depending, forexample, on the design constraints and the impact of edge effects at thesegmented electrode. Also, in the particular examples shown, theoctagonal shape of the layers 430 and 431 are elongated in orthogonaldirections to facilitate the attachment of drive electronics to thetransparent electrodes. In FIGS. 4 and 11, the octagonal anddifferential sizes of layers 430 and 431 are not shown (symmetricalround parts are shown) as the electronic requirements do not impact theoptically relevant features emphasized in these figures.

FIG. 8A is an example arrangement of electrodes usable for controllingthe rotator regions 436, 440. The transparent electrodes can be disposedon the glass layer 432 to impart appropriate polarization to the lightpassed through the two pupils 434 and 438 formed in the liquid crystallayer 424. The light that passes through the glass layer 432 ispolarized by the second polarizer 426. In some implementations, theglass layer 432 includes a radial pattern of electrodes such as theelectrodes 805 and 810. Neighboring electrodes are independent,electrically isolated from one another, and coupled to the electroniccircuitry configured to control the operations of the electronic pupils.The electrodes are driven by electronics conductively attached at theperimeter of the glass. In some implementations, the electrode segmentsdisposed on the glass layer 432 are larger in outer radius and smallerin inner radius in comparison to the electrode segments disposed in theglass layer 430 (as shown in FIG. 7A), to account for light that passesthrough the glass layer 430 at an angle. In some implementations, theconfiguration of the electrodes disposed in the glass layer 432 can besubstantially identical to the configuration of the electrodes disposedin the glass layer 430, at the potential cost of introducing vignettingon the inner and outer edges of the annular aperture segments. This maybe alleviated, for example, by using a mask 600 (FIG. 6A) that isradially undersized relative to the electrode array and using a mask 620(FIG. 6B) that is radially enlarged.

FIG. 8B is an example of candidate rotator segments created in theliquid crystal layer 428 with the combination of electrode arrangementsof FIGS. 7A, 7B, and 8A, 8C. The liquid crystal cells or amalgamation ofsegments 853 and 855, and the intervening liquid crystal layer form adevice that can rotate the polarization of incoming light into twoorthogonally polarized states, respectively. In some implementations,the segment 853 may polarize the light from one pupil in such a way thatthe light is orthogonally polarized with respect to the light from theother pupil passing through the segment 855. For example, the two groupsof segments (as depicted by two different shades) shown in FIG. 8B mayrepresent segments that rotate the polarization of the light by 90°, andsegments that let the light pass effectively unrotated, respectively.

FIG. 8C is an example of a transparent electrode that can be used inconjunction with the segments shown in 8A. The liquid crystal material428 (a typical twisted nematic (TN) type for example) can be disposedbetween the electrodes shown in FIGS. 8A and 8C. A portion of thetransparent electrode 864 can be configured to face the segments shownin FIG. 8A such that the portion faces the electrode areas 805, 810 toform the liquid crystal cells or segments (such as ones that may befound in the LCD display of a digital watch). While the example of FIG.8C (and those in other figures) show the electrode as a dark shadedportion, in practice the electrode layers can be substantiallytransparent, and constructed from a transparent conductive material suchas indium tin oxide (ITO). The active electrode area 864 can be drivenby electronics attached to the electrode, for example at locations 862and/or 863. The electronics may be interfaced at the locations 862, 863using various techniques such as ones using conductive adhesives,metallic clips, or zebra type connectors. By electronically controllingthe voltage on the electrode area 864 and the voltages on the electrodesof FIG. 8A, and creating controlled voltage differences between theelectrode area 864 and selected segments of FIG. 8A, patterns ofpolarization rotation can be imparted on the light traversing the liquidcrystal cell. Examples of segments supporting the patterns are shown inFIG. 8B.

Referring again to FIG. 8A, the electrode segments disposed on the glasslayer 432 have similar radial angles as that of the pupil formingelectrodes disposed in the layer 430 (as shown in FIG. 7A). However,other configurations are also possible. In some implementations, theelectrode segments disposed in the glass layer 432 may have a coarserresolution. For example, pupils that are known a-priori to be disposedat substantially 180° with respect to each other may be implementedusing only four electrode segments in the glass layer 432. This is shownusing the examples of FIGS. 9A, 9B, and 9C. Specifically, FIG. 9A isanother example arrangement of electrodes to form the four segments 902,904, 906, and 908. FIG. 9C shows an example of a transparent electrodeto complete the liquid crystal cells. The liquid crystal material 428 isinterposed between the electrodes shown in FIGS. 9A and 9C. Thetransparent electrode area 964 is disposed on the glass layer 433. Theelectrode is driven by connection through one or both of the locations962, 963, which connect to the drive electronics. Thus the segments 902,904, 906, and 908 when electrically controlled together with theelectrode area 964, can yield the pattern shown in FIG. 9B. In someimplementations, light passing through rotator segments 910 isorthogonally polarized to that passing through rotator segments 912.Thus, polarization control is achieved with the electrical manipulationof the first 424 and second 428 LCD layers. The electrode shown in FIG.9C is used in conjunction with those in FIG. 9A to produce the rotatorsegments illustrated in FIG. 9B. In the example of FIG. 9B, the rotatorsegments 910 impart one type of polarization while the rotator segments912 impart a different type of polarization (e.g., polarization that isorthogonal with respect to the polarization imparted by the rotatorsegments 910). In some implementations, the four rotator segmentsillustrated in FIG. 9B may be sufficient to maintain a 180° separationbetween the pupils 434 and 438. Naturally, other numbers of segments arepossible, and the examples shown are illustrative.

Referring again to FIG. 4, a surgeon may wish to be able to rotate theshaft of the 30° endoscope 400 to look at the sides of the cavity. Insuch cases, a pair of statically defined pupils will end up beinglocated one above the other, or in some other arbitrary position, wherethe horizontal separation needed for the corresponding stereo images islost. The technology described herein allows for dynamically definingthe pupils using the LCD segments such that the pupil locations adapt tothe rotation of the endoscopic camera 400. During a rotation, or arolling motion, the endoscope sees a different view relative to gravityand the pupils may be progressively adjusted as described to keep thecamera's left and right eyes separated correctly relative to thehorizon. Correspondingly, the images from the image sensors can beelectronically rotated to keep the orientation of the image displayedfrom the sensor oriented as desired on the display screen. In someimplementations, a circular image may be presented to the surgeon.

FIGS. 10A and 10B are examples of pupil positions for two differentorientations of a stereoscopic endoscope, respectively. In the examplesof these figures, the two pupils 434 and 438 are formed in the liquidcrystal layer 424 by energizing six LCD segments using correspondingelectrodes (FIG. 7A) defined on the glass layer 430 in conjunction withthe electrode 717 disposed on the glass layer 431. Specifically, thesegments 1010 a, 1010 b, and 1010 c (1010, in general) are energized todefine the pupil 438, and the segments 1015 a, 1015 b, and 1015 c (1015,in general) are energized to define the pupil 434. Note that while theenergized segments are shown in a dark shade, with the other segmentsbeing white, this is for illustration purposes only; as is known to onein the art, the liquid crystal appears transparent in both states andonly becomes ‘dark’ between appropriately oriented polarizers. Inpractice, the shaded segments would be energized such that they becomesubstantially transparent and allow light to pass through, while therest of the segments are held at the other state to hinder light frompassing through polarizer 426. Also, while the example of FIG. 10A showsthree segments per pupil, more or less number segments may be used foreach. Using more segments for each pupil would increase thecorresponding aperture's effective diameter (potentially improving lightthroughput) but reducing the effective inter pupillary distance, asmeasured by the distance between the centroids of the two regions 1010and 1015.

In the example of FIG. 10A, the stereoscopic endoscope is oriented at aparticular angle with respect to a reference orientation, as defined,for example, by the axis 1020 connecting the LCD segment 1010 b and 1015b. In some implementations, the axis 1020 may represent the “horizon”;i.e., an orientation that is perpendicular to the direction of theearth's gravity. However, other reference orientations are alsopossible. For example, the orientation of the surgeon's eyes relative toa large display mounted on the wall or a boom can be measured, and theposition the pupils of the endoscope may be adjusted substantiallysimilarly such that the surgeon's stereo perception remains accurate,even if the surgeon's head is tilted relative to the display. Making thepupil positions adaptive to the surgeon's head orientation mayparticularly improve the surgeon's user-experience in some cases; forexample, in laparoscopic surgery situations where the location for thesurgeon's hands may dictate a non-ideal body pose relative to thedisplay.

If the endoscope is rotated to another angle, such as in the exampleshown in FIG. 10B, the axial orientation of the endoscope changes withrespect to the reference orientation represented by the axis 1020. Insuch cases, to maintain the same angle of orientation, relative togravity, for the pupils 434, 438, as in the case of FIG. 10A, differentsets of LCD segments may need to be energized. In the example of FIG.10B, the LCD segments 1025 a, 1025 b, and 1025 c (1025, in general) areenergized to define the pupil 438, and the segments 1030 a, 1030 b, and1030 c (1030, in general) are energized to define the pupil 434.Therefore, to maintain the angle of orientation across multipleendoscope orientations, the aperture locations corresponding to thepupils are determined to maintain a predetermined spacing between thefirst and second aperture locations, and the apertures are then createdat the new locations by energizing the corresponding LCD segments. Whilethe examples in FIGS. 10A and 10B show a liquid crystal layer withradial LCD segments, other arrangements of such segments are alsopossible. Additionally, the predetermined distance requirement issomewhat arbitrary in the general case and could be changed in systemsusing the patterns shown in FIGS. 7A, 7B and 8A, 8C. The pattern shownin FIGS. 9A and 9C illustrates a 180 degree separation.

In some implementations, it may be desirable to be able to change thedistance between the pupils and change their relative orientation. FIG.10C illustrates two pupils formed by controlling the segments in the twoLCD layers. In this case, the pupils are not separated by 180 degrees.In this case, segments 1040 a, 1040 b, 1040 c form one pupil andsegments 1050 a, 1050 b, 1050 c form the second. These pupils, by thedescribed operation of the LCD layers are orthogonally polarized andwould form images, in the configuration of FIG. 4, on sensors 402 and404. Pupil separation and size may be controlled, FIG. 10D shows thepupils moved and changed in size. 10D shows pupils 1070 and 1080 whichare orthogonally polarized and in a different configuration as comparedto that shown in FIG. 10C. Pupils may be moved in orientation and orrelative position based on control signals which come from controlsignals (e.g., physical or in software) calculated from the imagesacquired by the endoscope or some other source. For example, the pupilspacing might be dynamically controlled based on the distance from theendoscope to the surgical site for example.

While the description above primarily uses the example of the 30°endoscope 400 shown in FIG. 4, the technology may also be used for otherendoscopes such as a 0° endoscope. FIG. 11 is a schematic diagram of anexample of a 0° endoscopic camera 1100, which includes two sensors 1102and 1104. The endoscope 1100 may be substantially similar to theendoscope 400 described with reference to FIG. 4, except that the 0°endoscope includes the lens arrangement 1105 instead of the 30° prism ofthe endoscope 400. The technology described above with reference to the30° endoscope 400 can be extended for the endoscope 1100 to control thelocations of the pupils 1112 and 1114 such that the correspondingcaptured light can be separated using the polarizing beam splitter 1106with appropriate coating or wire grid polarizing layer on surface 1110separating the two orthogonal polarizations so as to capture the stereoimages using the sensors 1104 and 1102, respectively. The selection ofwhich pupils are used allows the 0 degree endoscope to rotateelectronically without using moving parts—for example, the pupils can berotated as described, and the images rotated electronically (or insoftware). From the user's perspective, this can have the effect ofappearing to rotate the stereo camera system. This ability could also beused by machine vision algorithms for calculating characteristics of thesurface being observed, calculating distance, and surface normals forexample.

In some implementations, the electronic pupil control described hereinmay also be used to capture stereo images sequentially, using a singleimage sensor. For example, the two pupils corresponding to the leftimage and right image may be created sequentially, and the correspondingimages may be captured using a single sensor in a time-divisionmultiplexed arrangement. FIG. 12A is a schematic diagram illustrating anexample of a 0° stereoscopic endoscopic camera 1200 that uses one sensorfor sensing both stereoscopic images. In this approach, light from asurgical scene 1202 enters the endoscope 1200 through the lensarrangement 1105. In some implementations, the lens arrangement 1105 caninclude a first polarizer to polarize the incoming light. The opticalpath of the light through the endoscope 1200 is represented by examplesample rays 1206 passing through the right pupil 1207 and example rays1208 passing through the left pupil 1209. In some implementations, theglass layer 1210 can be thicker as compared to the corresponding layer430 in the endoscope 400 (FIG. 4) to support thinner cell geometries forferroelectric liquid crystal materials, which may be used for fastswitching of LCD segments defining the pupils. In some cases, the thickglass layer 1210 provides more stability and stiffer mechanical support,making it potentially easier to create and maintain thin cells for suchmaterials. In some implementations, the ferroelectric liquid crystalmaterial can be selected to facilitate fast switching (e.g., once every5 ms, 2 ms; 100 μs, or less).

The ferroelectric crystal material can be disposed as a thin layer(e.g., of width of a few μm) between active electrodes disposed on glass(or other transparent substrate) layers. In some implementations, theferroelectric liquid crystal layer can be configured to let linearlypolarized light pass through effectively unchanged, or, responsive toelectronic control, be rotated by 90°. In some implementations, thelight passing through the first pupil may be polarized differently fromthe light passing through the second pupil, for example, as describedabove with reference to FIG. 4. In some implementations, light from boththe left and the right pupils are polarized the same way and aretransmitted by a second polarizer. Light passing through the secondpolarizer then passes through optics 1212, which unlike the endoscope1100 (FIG. 11), does not include a beam splitter 1106. Rather the lightfrom the right pupil (represented by the rays 1214) as well as the leftpupil (represented by the rays 1216) pass through the optics 1212 toreach the image sensor 1218. In some implementations, pad printed blackmasks or some other light blocking elements may be incorporated in theoptical path, for example, to help improve the contrast ratio of theimage sensed by the image sensor.

In implementations where the light from both the right pixel and theleft pixel are sensed by the same sensor, the two corresponding imagescan be sensed in different ways. FIG. 12A shows an implementation wherethe light from the two pupils are polarized in different ways, andcorresponding polarizers can be used in the image sensor 1218 todifferentiate between the corresponding images. FIG. 12A shows anexpanded version of a portion 1220 of an example image sensor 1218. Theportion 1220 includes multiple pixels wherein a polarizer is overlaid oneach pixel. The polarizers for the pixels in a first set are differentfrom the polarizers for the pixels in a second set, such that the pixelsin the first and second set can selectively sense light withcorresponding polarization states. In the example shown in FIG. 12A, thepolarizer for pixel 1224 is orthogonal with respect to the polarizer forthe pixel 1226. Therefore, the image sensor can be configured to senselight from the two pupils simultaneously if the light from the pupilsare appropriately polarized and match the pixelated polarizer at theimage sensor 1218.

In the example of FIG. 12A, a subset of the pixels of the sensor 1218sense the light from the right pupil, and a different subset of thepixels sense the light from the left pupil. Therefore, the spatialresolution of each eye image is less (half) than that afforded by thesensor 1218. In some implementations, the full spatial resolution of theimage sensor 1218 can be used to sense the two images by using thesensor for each pupil sequentially on a time division multiplexed basis.This is illustrated in FIG. 12B using the example of a 30° endoscopewith one image sensor 1260. In this example, the electronic circuitrycontrolling the state of the pupils 1252 and 1254 are synchronized, forexample, with the timing signals controlling the frame rate for thesensor 1260. In the case of a sensor with a global shutter, which pupilis active alternates at each frame boundary. Thus, the left pupil 1252is open (and the right pupil 1254 is dark) when a left image is beingacquired. Similarly, the right pupil 1254 is open (and the left pupil1252 is dark) when a right image is being acquired. This sequence can berepeated to acquire stereo images at the spatial resolution afforded bythe sensor 1260, but at the cost of a lower frame rate for the stereoimages. For example, if the imager frame rate is 60 Hz, then a stereopair is acquired at 30 Hz although a ‘new’ image is acquired at the rateof 60 Hz. In another example, if the imager is run at a frame rate of120 Hz, a stream of stereo images at 60 Hz can be generated.

In some implementations, if the sensor 1260 employs a rolling shutter,the exposure time and readout for the two pupils may overlap, and thusneed to be accounted for. An example of such a scenario is illustratedin FIG. 13. In the example shown, each pupil is ‘open’ for twoconsecutive time units synchronized with the frame boundaries. Duringthe second of those two time units (each of which may also be referredto as a frame time), the data corresponding to the pupil with the openshutter is acquired. The readouts marked ‘imperfect’ in FIG. 13 areimages where the illumination on the sensor is not correct, and hencethe data corresponding to the readouts are discarded. Therefore, therate of acquisition of the stereo pairs in such an arrangement is onefourth of the underlying frame rate. For example, if the frame rate is240 Hz, stereo pairs are acquired at 60 Hz. In some implementations,because the data acquisition for one pupil is complete before that forthe other pupil, a sequential display arrangement can be used to reduceend-to-end latency of the system. In some implementations, if thesurgeon's head is moving, for a 240 Hz frame rate, the pupil positionscan be updated at 120 Hz rather than 60 Hz, to potentially improve thesystem response under fast head motion. For example, the images for leftand right pupils may be acquired at subsequent moments in time and assuch, the pupils may be placed in the ideal positions for each eye. Insome implementations, the pipelined nature of the arrangementillustrated in FIG. 13 provides reduced latency and resource relatedadvantages due to frames being processed sequentially rather thansimultaneously.

FIG. 14 is a flowchart of an example process 1400 for generating a 3Drepresentation using technology described herein. In someimplementations, operations of the process 1400 can be executed, atleast in part, by the processing device 43 described above withreference to FIG. 2. In some implementations, operations of the process1400 may be executed by one or more processing devices of an endoscopesuch as the endoscopes 400, 1100, 1200, or 1250 described above.Operations of the process 1400 includes determining an angle oforientation defined by a line connecting a first aperture location and asecond aperture location of a stereoscopic endoscope with respect to areference orientation (1410). In some implementations, the referenceorientation can be defined as the “horizon” that is perpendicular to thedirection of earth's gravity. In some implementations, the lineconnecting the first aperture location and the second aperture locationcan be substantially similar to the axis 1020 shown in FIG. 10A aspassing through the locations corresponding to the LCD segments 1010 band 1015 b. In some implementations, determining the referenceorientation can include receiving information indicative of anorientation of the head of a user operating a stereoscopic endoscope,and determining the reference orientation in accordance with theinformation indicative of the orientation of the head of the user;and/or relative to a display on a wall or boom.

Operations of the process 1400 also includes adjusting at least one ofthe first and second aperture locations, while maintaining a spacingbetween the first and second aperture locations, to maintain the angleof orientation across multiple endoscope orientations (1420). This caninclude, for example, selecting locations of a pair of liquid crystaldisplay (LCD) segments from a set of LCD segments disposed in asubstantially annular configuration in an optical path of thestereoscopic endoscope. In some implementations, the annularconfiguration of the LCD segments can be substantially as shown in theexample of FIG. 8B.

Operations of the process 1400 further includes creating an aperture ateach of the first and second aperture locations (1430). In someimplementations, this can be done, for example, using the combination ofelectrode arrays described above with reference to FIGS. 7A, 7B, and 8A.For example, creating the apertures can include controlling a first LCDsegment in the pair of LCD segments such that the first LCD segmentchanges to a state in which the first LCD element allows more light topass through as compared to a different, relatively minimallytransmissive or dark state, and controlling a second LCD segment in thepair of LCD segments such that the second LCD segment changes to a statein which the second LCD segment allows more light to pass through ascompared to a different, relatively minimally transmissive or darkstate. In some implementations, the apertures may be created in asequential pattern, as illustrated, for example, in FIG. 13.

Operations of the process 1400 also includes generating the 3Drepresentation for presentation on a display device associated with thestereoscopic endoscope using signals based on light captured through theapertures created at the first and second aperture locations (1440). Insome implementations, the display device can be substantially similar toa display device associated with the surgeon's console 50, as describedabove with reference to FIG. 2. In some cases, the process 1400 can alsoinclude receiving user input responsive to presenting the visualrepresentation of the surgical scene on the display device. For example,the user input can pertain to operating a surgical device (such as therobotic manipulator arm assembly 120 described with reference to FIG. 3)at the surgical scene. In some implementations, the signals cancorrespond to a first image and a second image acquired substantiallyconcurrently. In such concurrent acquisition, the light passing throughthe first LCD segment can be configured to pass through a firstpolarizer, and the light passing through the second LCD segment can beconfigured to pass through a second polarizer that polarizes lightdifferently from the first polarizer. In some implementations, the firstpolarizer can be substantially orthogonal to the second polarizer.

In some implementations, the light passing through the apertures createdat the first and second aperture locations is sensed using a firstsensor and a second sensor, respectively. The first and second sensorscan be disposed on two opposing sides of a polarizing beam splitter,such as in the arrangement described above with reference to FIG. 4. Insome implementations, the light passing through the apertures created atthe first and second aperture locations are sensed using a singlesensor, such as in the arrangements described above with reference toFIGS. 12A and 12B.

The functionality of the tele-operated surgery system described herein,or portions thereof, and its various modifications (hereinafter “thefunctions”) can be implemented, at least in part, via a computer programproduct, e.g., a computer program tangibly embodied in an informationcarrier, such as one or more non-transitory machine-readable media orstorage device, for execution by, or to control the operation of, one ormore data processing apparatus, e.g., a programmable processor, a DSP, amicrocontroller, a computer, multiple computers, and/or programmablelogic components.

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program can be deployed to be executed on one or moreprocessing devices at one site, or distributed across multiple sites andinterconnected by a network.

Actions associated with implementing all or part of the functions can beperformed by one or more programmable processors or processing devicesexecuting one or more computer programs to perform the functions of theprocesses described herein. All or part of the functions can beimplemented as, special purpose logic circuitry, e.g., an FPGA and/or anASIC (application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Components of a computer include aprocessor for executing instructions and one or more memory devices forstoring instructions and data.

While this specification contains many specific implementation details,these should not be construed as limitations on what may be claimed, butrather as descriptions of features that may be specific to particularembodiments. Other embodiments may also be within the scope of thetechnology described herein. For example, while the technology has beendescribed with reference to two mirrors and a single mirror bounce, thetechnology may be extended to any odd number of mirror bounces withoutdeviating from the scope of this disclosure. Certain features that aredescribed in this specification in the context of separate embodimentscan also be implemented in combination in a single embodiment.Conversely, various features that are described in the context of asingle embodiment can also be implemented in multiple embodimentsseparately or in any suitable subcombination. Moreover, althoughfeatures may be described herein as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can, in some cases, be removed from the combination, and theclaimed combination may be directed to a subcombination or variation ofa subcombination.

Elements of different implementations described herein may be combinedto form other embodiments not specifically set forth above. Elements maybe left out of the structures described herein without adverselyaffecting their operation. Furthermore, various separate elements may becombined into one or more individual elements to perform the functionsdescribed herein.

What is claimed is:
 1. A stereoscopic endoscope comprising: at least oneimage sensor for sensing a first image and a second image of a pair ofstereo images, the first image being sensed based on light passingthrough a first aperture within the stereoscopic endoscope, and thesecond image being sensed based on light passing through a secondaperture within the stereoscopic endoscope; a liquid crystal layerdisposed between two layers of glass comprising a first arrangement ofelectrodes, such that each of the first aperture and the second apertureis created in the liquid crystal layer using a portion of the firstarrangement of electrodes; a first portion housing a front end lensassembly; and a second portion comprising an elongated shaft that housesthe liquid crystal layer and the at least one image sensor, wherein asize of each of the first aperture and the second aperture, a spacingbetween the first aperture and the second aperture, and a polarizationstate associated with each of the first and second apertures arecontrolled using corresponding control signals provided through thefirst arrangement of electrodes.
 2. The stereoscopic endoscope of claim1, wherein the light passing through the first aperture is polarizeddifferently as compared to the light passing through the secondaperture.
 3. The stereoscopic endoscope of claim 2, comprising: a firstimage sensor; a second image sensor; and an optical element that directsincident light to the first image sensor or the second image sensorbased on polarization state of the incident light.
 4. The stereoscopicendoscope of claim 2, wherein a polarizer associated with the firstaperture is substantially orthogonal with respect to the secondaperture.
 5. The stereoscopic endoscope of claim 1, wherein locations ofthe first and second apertures are controllable in accordance with anorientation of the stereoscopic endoscope with respect to a referenceorientation.
 6. The stereoscopic endoscope of claim 1, comprising afirst image sensor and a second image sensor, wherein the first imageand the second image are sensed by the first image sensor and the secondimage sensor, respectively, substantially concurrently.
 7. Thestereoscopic endoscope of claim 1 wherein the first image and the secondimage are sensed by a single image sensor sequentially.
 8. Thestereoscopic endoscope of claim 1, wherein a location of at least one ofthe first and second apertures is electronically adjusted using thefirst arrangement of electrodes to maintain an angle between (i) a lineconnecting the first and second apertures, and (ii) a referenceorientation.
 9. The stereoscopic endoscope of claim 8, wherein the anglebetween (i) the line connecting the first and second apertures, and (ii)the reference orientation is maintained while also maintaining a spacingbetween the first and second apertures.
 10. The stereoscopic endoscopeof claim 8, wherein the angle between (i) a line connecting the firstand second apertures, and (ii) the reference orientation is maintainedusing one or more control signal calculated based on one or morepreviously captured images.
 11. The stereoscopic endoscope of claim 8,comprising one or more sensors for generating information indicative ofan orientation of the head of a user operating the stereoscopicendoscope.
 12. The stereoscopic endoscope of claim 11, comprising one ormore processing devices for determining the reference orientation inaccordance with the information indicative of the orientation of thehead of the user.
 13. The stereoscopic endoscope of claim 1, wherein theliquid crystal layer comprises a plurality of regions that areselectively switchable between a first transmissive mode, and a secondrelatively less transmissive mode, and wherein the first aperture andthe second aperture are created by energizing portions of the pluralityof regions using the portion of the first arrangement of electrodes. 14.The stereoscopic endoscope of claim 1, wherein the liquid crystal layercomprises a plurality of liquid crystal segments.
 15. The stereoscopicendoscope of claim 14, wherein the size of each of the first apertureand the second aperture is defined as a subset of the plurality ofliquid crystal segments.
 16. The stereoscopic endoscope of claim 1,wherein the light passing through a first aperture is captured using afirst image sensor, and the light passing through the second aperture iscaptured using a second image sensor, the first and second image sensorsbeing disposed on two sides of a polarizing beam splitter.
 17. Thestereoscopic endoscope of claim 1, wherein the spacing is controlledbased on a distance between the stereoscopic endoscope and a surgicalsite.
 18. The stereoscopic endoscope of claim 1, wherein the secondportion is disposed at a non-zero angle with respect to the firstportion.
 19. The stereoscopic endoscope of claim 18, wherein thenon-zero angle is one of: 30 °, 45°, and 70°.
 20. The stereoscopicendoscope of claim 18, further comprising a prism comprising a firstprismatic portion and a second prismatic portion arranged to transmitlight from the first portion of the stereoscopic endoscope to the secondportion of the stereoscopic endoscope.
 21. The stereoscopic endoscope ofclaim 20 further comprising an air gap at an interface between the firstprismatic portion and the second prismatic portion, the air gap arrangedto facilitate total internal reflection of at least a portion of thelight within the first prismatic portion for transmission the secondportion of the stereoscopic endoscope disposed at the non-zero anglewith respect to the first portion.